Reference is made to commonly assigned copending application Ser. No. 09/048,605, entitled xe2x80x9cDigital Photography System Using Direct Input to Output Pixel Mapping and Resizingxe2x80x9d and filed on common date herewith in the names of Kenneth A. Parulski and James E. Adams, Jr., and which is assigned to the assignee of this application.
The invention relates generally to the field of electronic photography, and in particular to an electronic imaging system of the type that generates a standard format file.
A typical digital camera uses a single color image sensor to capture a representation of a scene, including a representation of the color in the scene. The colors are separated by a color filter array(CFA) that overlies the sensor and associates each image pixel with a particular color according to the particular color pattern of the CFA. For example, the Bayer CFA (described in U.S. Pat. No. 3,971,065) provides an RGB pattern with a checkerboard arrangement of green pixel sites. Therefore, the primary CFA image signal for each pixel relates to but one color, that is, to either red, green or blue (for a Bayer CFA). After the CFA image is captured, the remaining (missing) colors for each pixel site are interpolated for each pixel from, e.g., surrounding pixels, so that a full resolution interpolated record is generated for each image. Each pixel in the interpolated record therefore has a set of RGB values.
Digital cameras, such as the Kodak DC50 camera, store the compressed digital image data from a single-chip color sensor (approximately 750xc3x97500 pixels) on a removable memory card. The process used to convert the image to a standard file format, and to print and edit the standard format, is shown in FIG. 1. The CFA image is captured in a capture step 10 by the camera. When the camera or memory is connected to a host computer in a connect step 12, the images are downloaded from the camera or memory card in a downloading step 14, processed in a processing step 16 and stored in a standard file format, such as a FlashPix file format, in a file formatting step 18. In the processing step 16, the CFA images are CFA interpolated as they are imported, using known CFA interpolation, color correction, and sharpening techniques, in order to create a xe2x80x9cfinishedxe2x80x9d RGB image file having a standard size. The finished RGB image file thus contains the CFA interpolated image, which has the same number of total pixels as the original image (for example, approximately 750xc3x97500 pixels for the DC50 camera). Unlike the original image, however, the CFA interpolated image has complete RGB data for each pixel.
The advantage of using a standard file format is that it allows the image to be used by many different imaging-enabled programs supplied by many different companies. Many different file formats are known and useful with this invention, albeit the following description is mainly in terms of a FlashPix file. FIG. 2 is a simplified diagram showing the key information stored in a FlashPix file. The FlashPix image format (defined in FlashPix Format Specification, version 1.0.1, (Digital Imaging Group, Jul. 10, 1997)) has been developed to serve as both an xe2x80x9cinterchangexe2x80x9d format between devices (e.g. cameras) and applications (e.g. computer picture editing packages), and as a xe2x80x9cnativexe2x80x9d format for image editing that allows the images to be easily and rapidly edited. This is accomplished using a hierarchical, tiled image representation within a xe2x80x9cstructured storagexe2x80x9d file. Referring to FIG. 2, a FlashPix file contains the complete image data 24 plus a hierarchy of several lower-resolution copies within the same file (one set of lower resolution image data 25 is shown in FIG. 2).
Images at each resolution also are divided into rectangular tiles (e.g., squares), which enable the application to minimize the amount of image data processed to access, display or print a portion of the scene content. FlashPix allows thumbnail image data 23 and optional xe2x80x9cancillaryxe2x80x9d property set data 21 and 22 to be stored in the same structured storage file, along with the image data. This ancillary data can include digital audio recordings and parameters such as the time and date the photo was taken, the camera zoom position and focus distance, the scene illumination level, camera calibration data, image copyright owner, etc. For further information about the FlashPix image format see the aforementioned FlashPix Format Specification, version 1.0.1, (Digital Imaging Group, Jul. 10, 1997), which is available on the Wide World Web at http://www.digitalimaging.org/go/flashpix, and is incorporated herein by reference.
The image processing used to xe2x80x9cfinishxe2x80x9d the image prior to storing the finished data in the FlashPix file includes operations such as CFA interpolation, color correction, and image sharpening. The output images are normally stored in a standard color space, such as the sRGB space which is supported by the FlashPix image format. As a consequence of generating the image file, the original camera data is discarded (step 26). If an image is to be printed or transmitted in an application step 28, the standard file format is first edited in an editing step 27.
To create final printed images, the finished image file may be adjusted by the user via an image processing program, such as provided by a program like LivePix(trademark) by Live Picture, Inc., to create a final output image of the desired size, which may include only a xe2x80x9ccroppedxe2x80x9d, color adjusted portion of the image captured by the camera, or may, e.g., include multiple images in a xe2x80x9cmontagexe2x80x9d. More specifically, to provide the appropriately sized output image, the image processing program first crops the 750xc3x97500 RGB CFA interpolated image data, and then uses another interpolation algorithm in the editing step 27 to convert from the cropped version of the xe2x80x9cfinishedxe2x80x9d RGB image file to the final desired output image.
Note that this prior art approach uses two interpolation steps, one (step 16) to interpolate xe2x80x9cmissingxe2x80x9d RGB data from the image sensor while maintaining the 750xc3x97500 pixel data array, and a second (step 27) to interpolate from the selected crop to the pixel data array that provides the desired output size. The user therefore selects the final print size (for example 8xe2x80x3xc3x9710xe2x80x3), and the image is interpolated up to produce an image data record of appropriate size for the printer. For example, if the output device printed at 200 pixels per inch, the image data would be interpolated to 1600xc3x972000 pixels by the computer or the printer, (and optionally sharpened) to produce the desired image size. In addition, this known approach often uses two separate sharpening operations, one on the 750xc3x97500 pixel CFA interpolated image data following the CFA interpolation step 16, and a second one on the interpolated 1600xc3x972000 pixel image data array just prior to printing.
The prior art approach has a number of problems. First, the process of converting the image from the CFA image data (which may for example produce 12 bit per color data) to the 8 bit per color (24 bit per pixel) sRGB color space may limit the quality of certain images, by reducing the color gamut and dynamic range of the captured image. Second, the two interpolation steps (the CFA interpolation that produces a standard size image and the interpolation in the computer or printer that produces the desired print size) provide more interpolation artifacts than would be produced using a single interpolation step that interpolates from the sensor CFA data directly to the output image data sent to the printer. Finally, using two separate sharpening steps can also produce artifacts.
What is needed is a digital camera system that maintains the advantages of using a standard, xe2x80x9cfinishedxe2x80x9d image file format so that the images can be used by many applications, yet also enables image processing from raw camera data to final output data to be completed in a single, integrated process, to provide improved image quality when printing.
The present invention is directed to overcoming one or more of the problems set forth above. Briefly summarized, according to one aspect of the present invention, an electronic still imaging system employs an image sensor comprised of discrete light sensitive picture elements overlaid with a color filter array (CFA) pattern to produce color image data corresponding to the CFA pattern, an A/D converter for producing digital CFA image data from the color image data, and a memory for storing the digital CFA image data from the picture elements. A processor enables the processing of the digital CFA image data to produce finished image data, and the digital CFA image data and the finished image data are both stored together in an image file.
With regard to a specific embodiment, the xe2x80x9coriginalxe2x80x9d image data (which is optionally compressed using a numerically lossless or visually lossless technique) is stored in a digital image file on a digital memory and transferred to a host computer. On the host computer, the digital camera data is processed, and a xe2x80x9cFlashPixxe2x80x9d format image file (or some other xe2x80x9cfinishedxe2x80x9d file format) is created. The original camera image data is also stored within the FlashPix file, in an xe2x80x9cextensionxe2x80x9d. The extension also includes data that indicates whether the FlashPix image data has been modified, rather than simply adjusting the displayed FlashPix image using the xe2x80x9cviewing parameters.xe2x80x9d
The advantage is that the FlashPix image file, which can be used/edited by any FlashPix enabled application software, just like a standard FlashPix file, can also be used for direct pixel processing in order to minimize interpolation and other processing artifacts. To create the final high quality hard copy image, the (possibly cropped portion of the) original pixel data from the extension is directly interpolated (and optionally sharpened) in a single step to create the properly sized and color corrected output image. If the printer is located remotely, only the original camera data contained in the FlashPix extension is transmitted to the remote printer, thus reducing the transmission time.
These and other aspects, objects, features and advantages of the present invention will be more clearly understood and appreciated from a review of the following detailed description of the preferred embodiments and appended claims, and by reference to the accompanying drawings.